Immediate-mode camera for portable personal electronic devices

ABSTRACT

Techniques are described for implementing an immediate-mode camera for integration into portable personal electronic device (PPED) environments. Embodiments of the IMC can include an integrated digital camera that can be triggered directly by one or more user interface components, without involving waking up the application processor, waking up display components, waking up digital camera components, and/or starting up camera-related applications. For example, if a user&#39;s smart phone is locked, and the user desired to capture a photo, the user can interact with particular UI components in a particular manner, thereby directly triggering capture of image data by the IMC substantially without delay. Implementations of the IMC can involve a low-power, always-on digital camera that is directly controllable by one or more always-on user interface components. The IMC components can be in communication with an always-on region of the application processor via a fast hardware interface.

FIELD

The invention relates generally to cameras integrated into personalelectronic devices. More particularly, embodiments relate to animmediate-mode camera for integration into portable personal electronicdevice environments.

BACKGROUND

In the past, photography was a discipline reserved to those withspecialized knowledge and equipment. Over the past decades, innovationsin digital photographic hardware and software, and the worldwide spreadof smartphones with integrated digital cameras, have placed digitalphotography at the fingertips of billions of consumers. In thisenvironment of ubiquitous access to digital photography and videography,consumers increasingly desire to be able to quickly and easily capturemoments using their smartphones. However, the camera, and othercomponents of the smartphone (e.g., display hardware, illuminationhardware, etc.) can consume appreciable power if left in an active mode,which can appreciably reduce battery life and have other undesirableeffects. As such, smartphones (and some or all of their higher powercomponents) are typically kept in a sleep mode when not in use. Forexample, smartphones often are kept in a locked state, and most or allfunctions of the smartphone can only be accessed after unlocking thesmartphone.

When the smartphone is locked, and a consumer desires to capture a photoor video, the consumer typically presses a button, or interacts withanother interface, to unlock the smartphone. This wakes up certaincomponent resources, such as the smartphone display. A user interface ofthe smartphone can be used to access a camera application. When thecamera application starts, the application can access and wake upvarious camera-related component resources, such as camera andillumination hardware and software. After the camera-related componentresources are ready, the consumer can interact again with user interfaceelements to trigger the camera to capture a photo or video. For manytypical smartphone, there can be a delay of a couple to a few secondsbetween the time the smartphone is unlocked and the time when a photocan be captured by the smartphone. In some situations, such a delay istoo long. For example, a particular moment may have passed without timeto capture it. Thus, while maintaining the camera in a sleep mode whennot in use can help to appreciably increase battery life of thesmartphone, accessing and waking the camera from the sleep mode can addappreciable delay to use of the camera.

BRIEF SUMMARY OF THE INVENTION

Embodiments provide circuits, devices, and methods for implementing animmediate-mode camera for integration into portable personal electronicdevice (PPED) environments. Embodiments of the IMC can include anintegrated digital camera that can be triggered directly by one or moreuser interface components, without involving waking up the applicationprocessor, waking up display components, waking up digital cameracomponents, and/or starting up camera-related applications. For example,if a user's smart phone is locked, and the user desired to capture aphoto, the user can interact with particular UI components in aparticular manner, thereby directly triggering capture of image data bythe IMC substantially without delay. Implementations of the IMC caninvolve a low-power, always-on digital camera that is directlycontrollable by one or more always-on user interface components. The IMCcomponents can be in communication with an always-on region of theapplication processor via a fast hardware interface.

According to one set of embodiments, a portable personal electronicdevice (PPED) is provided for immediate-mode digital image capture. ThePPED includes: an application processor configured selectively tooperate in a sleep mode or an active mode, the application processorhaving an always-on region that remains active when the applicationprocessor is in the sleep mode; a plurality of user interface componentshaving one or more always-on user interface components, at least one ofthe one or more always-on user interface components being an IMC triggercomponent; and an immediate-mode camera (IMC) system having a digitalcamera assembly controlled by an IMC controller, the IMC controllercoupled with and responsive to the IMC trigger component. The IMCcontroller is configured to: detect an interaction with the IMC triggercomponent; trigger, in response to detecting the interaction, capture ofimage data by the digital camera assembly; pretreat the image data togenerate pretreated image signals; and communicate the pretreated imagesignals via an interface bus to the always-on region of the applicationprocessor.

According to another set of embodiments, a method is provided forimmediate-mode digital image capture. The method includes: detecting, byan immediate-mode camera (IMC) controller of an IMC system integratedinto a portable personal electronic device (PPED), an interaction withan IMC trigger component coupled with the IMC controller, the IMCtrigger component being one of a set of always-on user interfacecomponents integrated into the PPED; triggering, by the IMC controller,in response to the interaction, capture of image data by a digitalcamera of the IMC system; pretreating the image data by the IMCcontroller to generate pretreated image signals; and communicating thepretreated image signals via an interface bus to an always-on region ofan application processor integrated into the PPED.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, referred to herein and constituting a parthereof, illustrate embodiments of the disclosure. The drawings togetherwith the description serve to explain the principles of the invention.

FIG. 1A shows a simplified block diagram of a partial conventionalportable personal electronic device (PPED) architecture 100;

FIG. 1B shows a flow diagram of an illustrative conventional routine fortransitioning a PPED from sleep mode to being able to capture a photo onan integrated digital camera;

FIG. 2A shows a block diagram of a partial novel PPED architecturehaving an integrated immediate-mode camera (IMC), according to variousembodiments;

FIG. 2B shows a flow diagram of an illustrative routine fortransitioning the novel PPED of FIG. 2A from sleep mode to being able tocapture a image data using an IMC system, according to variousembodiments;

FIGS. 3A and 3B show front and side views, respectively, of anillustrative PPED, according to various embodiments;

FIGS. 4A-4C show illustrative implementations of PPEDs with varioustypes of IMC trigger components, according to various embodiments; and

FIG. 5 shows a flow diagram of an illustrative method for immediate-modedigital image capture, according to various embodiments.

In the appended figures, similar components and/or features can have thesame reference label. Further, various components of the same type canbe distinguished by following the reference label by a second label thatdistinguishes among the similar components. If only the first referencelabel is used in the specification, the description is applicable to anyone of the similar components having the same first reference labelirrespective of the second reference label.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, numerous specific details are provided fora thorough understanding of the present invention. However, it should beappreciated by those of skill in the art that the present invention maybe realized without one or more of these details. In other examples,features and techniques known in the art will not be described forpurposes of brevity.

It has become increasingly common for portable personal electronicdevices (PPEDs) to have various integrated hardware components thatconsume relatively large amounts of power when in operation, such as oneor more large displays, digital cameras, and illumination components. Tosave power and extend battery life, such PPEDs typically switch into asleep mode, in which the more power-hungry components are either turnedoff completely or are set to a low-power mode (e.g., standby power mode,or the like). As used herein, a PPED can include a smartphone, tabletcomputer, laptop computer, smart wearable device (e.g., a smartwatch),or any other suitable device that has one or more digital camerasintegrated therein. For example, when a conventional smartphone is“locked,” its camera (or each of its cameras) is effectively turned off.From that state, accessing the camera to take a picture conventionallyinvolves waking up the main processor (e.g., by unlocking thesmartphone) and accessing a camera application, which triggers thecamera also to wake up and prepare its subsystems (e.g., focus,illumination, etc.) for capture of images or video.

As used herein, a PPED can include a smartphone, tablet computer, laptopcomputer, smart wearable device (e.g., a smartwatch), or any othersuitable device that has one or more digital cameras integrated therein.It is assumed herein that a PPED can be set to a sleep mode. Differenttypes of PPEDs can be set thus in different ways, such as by usinghardware and/or software controls to lock the PPED, to set the PPED to“hibernate” or “sleep,” etc. As used herein, phrases like “locking,”setting or putting in “locked” mode, setting or putting in “sleep” mode,or the like are intended generally and interchangeably to refer toplacing the PPED, or one or more components of the PPED, in a mode inwhich one or more conventionally higher-power components (e.g., mainapplication processor 190, digital camera components 140, displaycomponents 125, etc.) are set to consume little or no power, such asstandby power. Similarly, as used herein, phrases like “unlocking,”setting or putting in “unlocked” mode, “waking,” or the like areintended generally and interchangeably to refer to returning the PPED,or one or more components of the PPED, to a mode in which the one ormore conventionally higher-power components are permitted to operate intheir conventionally higher-power manner.

For added context, FIG. 1A shows a simplified block diagram of a partialconventional portable personal electronic device (PPED) architecture100. As illustrated, such a conventional PPED can include multiplehardware components, such as hardware user interface (UI) components105, digital camera components 140, display components 125, andillumination components 135. For example, the UI components 105 caninclude physical buttons and switches, touch-sensitive components (e.g.,capacitive features) of a touchscreen, force sensor components, opticalsensor components, etc.; digital camera components 140 can includecharge-coupled devices (CCDs), lenses, etc.; display components 125 caninclude one or more display screens, etc.; and illumination components135 can include light-emitting diodes, etc. The hardware components canalso include memory 155, which can be implemented as one or moreintegrated memory hardware devices (e.g., removable and/or non-removablesolid-state memory circuits, etc.).

The conventional PPED can also include an application processor 190,which can be in communication with the hardware components via one ormore hardware interfaces 180. The application processor 190 can beimplemented as any suitable processor or processors, such as amicrocontroller unit (MCU), general-purpose processor, etc. Theapplication processor 190 can execute operating system functions, kernelfunctions, application-related functions, and any other suitablefunctions. Such functions can be used to control and/or respond tooperations of the hardware components. For example, outputting images tothe display 125 can involve the application processor 190 requestingsystem resources (e.g., memory resources) to allocate to the display,converting application-level commands to hardware-level commands, etc.

Conventionally, when a PPED is locked, certain portions of theapplication processor 190 and certain lower-power hardware componentscan continue to operate. Such components and functions that continue tooperate while the PPED is locked are referred to herein as “always-on”components or functions. Though referred to as “always-on,” suchcomponents or functions can be turned off in certain circumstances, forexample, by fully powering down the PPED, by removing a battery, etc. Insome PPEDs, options may be provided to permit selective shutting down ofsome or all “always-on” functions or components using hardware,software, etc. It is typical, however, for certain types of PPEDs to bekept in locked mode when not in use, as opposed to being fully shutdown. For example, when a smartphone is locked, it can still typicallyreceive phone calls, receive push notifications from applications, anddetect user interactions involved in unlocking the smartphone. As such,when the smartphone is locked, the application processor 190 may be setto standby power mode, or the like, in which the application processor190 draws a minimum amount of power to maintain operation of only its“always-on” functions; and low-power components (e.g., a fingerprintsensor, home key, or the like) can be maintained in an always-on stateto facilitate detection of unlocking interactions.

The digital camera components 140 are conventionally not always-oncomponents, as they tend to consume appreciable power and tend to becontrolled at the application layer by one or more camera-relatedapplications running on the application processor 190. As such, asdescribed above, when the PPED is locked, the digital camera components140 are typically in a sleep mode (e.g., turned off). Thus, accessingthe camera from sleep mode can involve waking the application processor190, accessing one or more applications, and waking digital cameracomponents 140 (e.g., and typically also waking the display components125 for viewfinder use, waking the illumination components 135 forlighting, etc.).

FIG. 1B shows a flow diagram 170 of an illustrative conventional routinefor transitioning a PPED from sleep mode to being able to capture aphoto on an integrated digital camera. For added clarity, the flowdiagram 170 is segregated into a user interface layer 161, anapplication layer 162, and a hardware layer 163. The user interfacelayer 161 includes portions of the flow diagram 170 involving userinteractions with hardware UI components 105. The application layer 162includes portions of the flow diagram 170 performed by the applicationprocessor 190 (e.g., executable by application software). The hardwarelayer 163 includes portions of the flow diagram 170 performed byhardware components other than the UI components 105 and the applicationprocessor 190, such as by the display components 125, the illuminationcomponents 135, the digital camera components 140, and the memory 155.As illustrated, in many conventional PPEDs, some or all of the hardwarecomponents are in communication with the application processor 190 viaone or more hardware interfaces 180 configured according to a MobileIndustry Processor Interface (MIPI) specification (shown as a MIPIinterface 145). For example, the digital camera components 140 can be incommunication with the application processor 190 via a MIPI CameraSerial Interface (MIPI CSI), the display components 125 can be incommunication with the application processor 190 via a MIPI DisplaySerial Interface (MIPI DSI), etc.

It is assumed that the flow diagram 170 begins when the PPED is locked,such that the digital camera components 140 (e.g., and othernot-always-on hardware components) are powered down, powered off, orotherwise in sleep mode. As such, the flow diagram 170 begins with aninteraction captured by an always-on component, resulting in a “wakeup”signal 107. To that end, a portion of the UI components 105 are shown asalways-on UI (AO-UI) components 105 a. For example, the AO-UI components105 a can include a button, fingerprint sensor, or other low-powercomponent configured to trigger the PPED to wake up in response todetecting certain interactions. A user interaction with one of the AO-UIcomponents 105 a generates the wakeup signal 107, which can trigger awakeup routine 110.

The wakeup routine 110 can involve any suitable commands to wake up thePPED, for example, beginning with commencing wakeup of the applicationprocessor 190. Typically, the wakeup routine 110 can also trigger adisplay on routine 120, by which the application processor 190 candirect the display components 125 to turn on, switch from a lock screento a home screen, and/or otherwise enter an operational mode for use bythe user. In some conventional PPED environments, a camera application130 is accessible via one or more UI components 105 after the displaycomponents 125 are turned on. For example, a number of icons aredisplayed by the display components 125 for selection by the user,including a particular icon associated with the camera application 130.User interaction with the particular icon, for example, by interfacingwith a touchscreen, navigation buttons, and/or other UI components 105,generates an access signal 127 for accessing the camera application 130.

Receipt of the access signal 127 by the application processor 190 cancause the camera application 130 to execute (e.g., to open). Executionof the camera application 130 can trigger the digital camera components140 to wake up (e.g., to power on, initialize, etc.). In some cases, thecamera application 130 can trigger one or more other systems to wake up,such as certain illumination components 135 (e.g., for providing flashand/or other illumination to support image or video capture), certainsensor components (e.g., for automatic stabilization), certain UIcomponents 105 (e.g., any buttons or other interfaces used only forcamera interactions), etc. The digital camera components 140 may finallybe used to capture image data (e.g., for photos and/or videos) after thedigital camera components 140 have been woken up (e.g., and, in somecases, only after waking up of any other components used by the cameraapplication). To take a photo or video, a user can typically interfacewith one or more UI components 105 to generate a capture signal 129.Receipt of the capture signal 129 by the application processor 190 cancause the camera application 130 to direct the digital camera components140 (and any other supporting components) to capture image data. Thecaptured image data may be pre-treated for communication to otherapplications via the MIPI interface 145. For example, pre-treated imagedata signals can be sent to a photo application 150 running on theapplication processor 190. The photo application 150 may generate photopreviews, interface with a photo storage and/or retrieval in the memory155, and/or provide other photo-related functions.

As described above, the digital camera components 140 conventionallyintegrated into PPEDs consume appreciable amounts of power, and thedigital camera components 140 are turned off when the PPED is locked tosave power (i.e., the digital camera components 140 are notconventionally implemented as always-on components). However, asillustrated by the flow diagram 170 of FIG. 1B, accessing the camerafrom sleep mode conventionally involves a number of stages, includingwaking and executing multiple hardware and software components. This canoften take multiple seconds (e.g., 1.5-3.5 seconds for many conventionalsmartphones, depending on processor speeds, camera resolution, and/orother specifications of the smartphones). Such a delay can beundesirably long for certain applications, such as when a user desiresto quickly capture a fleeting moment.

Some conventional PPEDs include a wakeup-to-camera mode, whichautomatically accesses the camera application 130 along with unlockingthe PPED. A particular interaction with an AO-UI component 105 a of thePPED while locked (e.g., swiping a finger across the lock screen along aparticular path, rotating the locked PPED back and forth in a particularmanner, etc.) causes the PPED to wake up and proceed automaticallywaking up the camera. For example, such a particular interactiongenerates both the wakeup signal 107 and the access signal 127, therebytriggering the wakeup routine 110, the display-on routine 120, andstart-up of the camera application 130; which in turn triggers wakeup ofthe display components 125, the illumination components 135, and thedigital camera components 140. While such a wakeup-to-camera mode can befaster than waiting for manual accessing of the camera application 130by the user, the wakeup-to-camera mode still involves the other portionsof the flow diagram 170 (e.g., waking up the application processor 190,waking up the display components 125, starting up the camera application130, waking up the digital camera components 140, etc.). As such, evenwith the wakeup-to-camera mode, there can still be an appreciable delaybetween the user's initial wakeup interaction with the PPED, and thedigital camera components 140 being ready for capturing photos orvideos.

Embodiments described herein include a novel immediate-mode camera (IMC)for integration in a PPED. Embodiments of the IMC can include anintegrated digital camera that can be triggered directly by one or moreuser interface components, without involving waking up the applicationprocessor, waking up display components, waking up digital cameracomponents, and/or starting up camera-related applications. For example,if a user's smart phone is locked, and the user desired to capture aphoto, the user can interact with particular UI components in aparticular manner, thereby directly triggering capture of image data bythe IMC substantially without delay. Implementations of the IMC caninvolve a low-power, always-on digital camera that is directlycontrollable by one or more always-on user interface components. The IMCcomponents can be in communication with an always-on region of theapplication processor via a fast hardware interface.

FIG. 2A shows a block diagram of a partial novel PPED architecture 200having an integrated immediate-mode camera 210, according to variousembodiments. As illustrated, the novel PPED architecture 200 can includemultiple hardware components, such as hardware user interface (UI)components 105 (including one or more always-on UI components 105 a),display components 125, and illumination components 135. The novel PPEDarchitecture 200 also includes an immediate-mode camera (IMC) system210. The IMC system 210 can include an integrated IMC microcontrollerunit (IMC MCU) 215, which can be directly controlled by one or more ofthe always-on UI components 105 a. Embodiments of the IMC system 210 caninclude an integrated IMC microcontroller unit (IMC MCU) 215, which canbe directly controlled by one or more of always-on components of the UIcomponents 105. The particular always-on components configured todirectly control the IMC MCU 215 are illustrated as one or more IMCtrigger components 205. Some such novel PPED architectures 200 alsoinclude one or more conventional digital camera components 140 (e.g., tooperate as separate digital cameras of the PPED, not operable asimmediate-mode cameras).

Embodiments of the IMC system 210 can be implemented as a low-powerdigital camera, such that the digital camera components can be always-oncomponents. The IMC system 210 includes one or more digital cameraassemblies. Embodiments of the digital camera assembly (or assemblies)can include any suitable components. For example, a photodetector ispositioned substantially at a focal plane of an imaging lens, such thatthe imaging lens forms an image of an object at the surface of thephotodetector. The photodetector can convert optical signals of theimage into electrical signals using, for example, a photodiode array, acharged coupled device (CCD), a complementary metal oxide semiconductor(CMOS) sensor, and/or any other suitable component.

Some embodiments of the IMC system 210 are packaged with a small openingwindow that can be integrated into a frame of a display screen of thePPED 300, or in any other suitable location. The IMC system 210 cameracan be in place of, or in addition to, one or more other digital camerasof the PPED 300. In some implementations, the IMC system 210 includesmultiple camera assemblies. Some embodiments of the IMC system 210include, or are configured to directly control, one or more of theillumination components 135. In one implementation, the IMC system 210includes one or more illumination light sources positioned in proximityto an aperture of the digital camera. The illumination light source(s)can provide illumination at one or more wavelengths, such as in anoptical wavelength (e.g., to provide illumination in a darkenvironment), in an infrared wavelength (e.g., at about 940 nmwavelength, or other eye-safe wavelength), etc.

The PPED architecture 200 can also include an application processor 190.The application processor 190 can be implemented as any suitableprocessor or processors. As illustrated, embodiments of the applicationprocessor 190 can include an always-on region 240. When the PPED islocked, the AO region 240 stays on (e.g., consuming relatively lowpower) to maintain operation of certain background functions. Forexample, while the PPED is locked, the AO region 240 of the applicationprocessor 190 can continue to monitor for user interactions withalways-on UI components 105, including IMC trigger components 205, todetect wake-up interactions, IMC-related interactions, etc. In someembodiments, the AO region 240 of the application processor 190 isimplemented in a trusted execution environment (TEE) 240.

Embodiments of the application processor 190 can be in communicationwith the hardware components (including the IMC system 210) via one ormore hardware interfaces 180. As described above with reference to aconventional PPED, some of the hardware components can be incommunication with the application processor 190 via a MIPI interface.For example, the conventional digital camera components 140 can be incommunication with the application processor 190 via a MIPI CSI, thedisplay components 125 can be in communication with the applicationprocessor 190 via a MIPI DSI, etc. The MIPI interface, and other similarinterfaces, facilitate use of the application processor 190 as aso-called “host processor” of the hardware component. For example, aconventional integrated digital camera does not have its ownmicrocontroller; instead, operation of the digital camera relies on theapplication processor 190 as its host processor and on communicationswith the application processor 190 via an appropriate interface, such asa MIPI interface. While such implementations provide a number ofdesirable features, MIPI-types of interfaces can be slow relative toother types of interfaces, such as processor-to-processor, bus-typeinterfaces. In some embodiments, the IMC system 210 is in communicationwith the application processor 190 via a such a faster bus-typeinterface, such as a Serial Peripheral Interface (SPI) 235. The SPI 235can operate as a communication interface bus coupled between theapplication processor 190 and the IMC system 210 to enable rapidsynchronous serial communications, for example, between the AO region240 of the application processor 190 and the IMC MCU 215 of the IMCsystem 210.

As described further herein, a user interaction with an IMC triggercomponent 205 can directly control the IMC MCU 215 to cause the IMCsystem 210 to capture image data. The captured image data can bepretreated by the IMC system 210 and sent, via the SPI 235, to the AOregion 240 of the application processor 190. Thus, photos and/or videoscan be captured by the IMC system 210 of the PPED without waiting for(e.g., and, in some cases, without even triggering) waking of theapplication processor 190, waking of display components 125, waking ofconventional digital camera components 140, starting up of applications,etc.

FIG. 2B shows a flow diagram 220 of an illustrative routine fortransitioning the novel PPED 200 of FIG. 2A from sleep mode to beingable to capture a image data using the IMC system 210, according tovarious embodiments. For added clarity, the flow diagram 220 issegregated into a user interface layer 161, an application layer 162,and a hardware layer 163. The user interface layer 161 includes portionsof the flow diagram 220 involving user interactions with hardware UIcomponents 105. The application layer 162 includes portions of the flowdiagram 220 performed by the application processor 190 (e.g., executableby application software). The hardware layer 163 includes portions ofthe flow diagram 220 performed by hardware components other than the UIcomponents 105 and the application processor 190.

It is assumed that the flow diagram 220 begins when the PPED 200 islocked, such that any components that are not always-on hardwarecomponents are powered down, powered off, or otherwise in sleep mode. Asdescribed herein, in accordance with the novel PPED architecture, theIMC system 210 and one or more IMC trigger components 205 areimplemented as always-on components. As such, the flow diagram 220 canbegin with an interaction captured by one of the IMC trigger components205, and the interaction can directly generate a capture signal 129 totrigger image and/or video capture by the IMC system 210. In someembodiments, the captured image data can be pretreated for communicationto the application processor 190. For example, the signals can bepretreated for communication, via the SPI 235, to the AO region 240 ofthe application processor 190. An application running in the AO region240 can process the received data and can interface with memory 150, asneeded.

For example, suppose an individual is carrying a locked smartphone andsees something that the individual wishes to quickly capture on camera.With a typical conventional smartphone, the individual would (a)interface with the smartphone to unlock the smartphone, (b) wait for thedisplay to wake up and become active, (c) interface again with thesmartphone to locate and select a camera application icon, (d) wait forthe camera application to startup and wake up the camera components, and(e) interface yet again with the smartphone to point the camera in thedesired direction and trigger image capture via the camera application.With the novel PPED architecture described herein, the individual cansimply point the IMC system 210 camera in the desired direction andinterface with the IMC trigger components 205 to trigger image capture.

As illustrated, some embodiments permit additional functions to betriggered by interaction with the IMC trigger components 205. In somesuch embodiments, while the IMC system 210 is capturing image data, someimplementations can, in parallel, begin to wake up sleeping portions ofthe application processor 190, display components 125, illuminationcomponents 135, etc. For example, when an individual first captures animage (or begins to capture a video) with the IMC system 210 byinterfacing with an IMC trigger component 205, the individual may beoperating “blind”; the individual may be capturing the image or videowithout visual feedback, for example, from a viewfinder. In themeantime, however, the application processor 190 and display components125 are waking up, such that visual feedback begins to be availableshortly thereafter. In such implementations, for example, a firstpicture (or first few pictures in rapid succession, or first second ortwo of a video) may be captured without visual feedback assistance;after which the individual can continue capturing images or video withvisual feedback assistance (e.g., thereby permitting the individual todynamically correct pointing, lighting, zoom, etc.).

In some embodiments, the IMC MCU 215 dynamically controls capture ofimage data by the IMC system 210 in response to the IMC triggercomponents 205. For example, the IMC MCU 215 can control operation ofoptical components for focus, lighting, tracking, image stabilization,and/or other image capture settings. For example, the IMC MCU 215 candetect lighting, moving objects and/or objects of interest in the fieldof view, and/or the like, and can dynamically adjust image capturesettings, accordingly. Additionally or alternatively, some embodimentsoperate according to default settings and/or user-configurable imagecapture settings. For example, a user can configure IMC settings, suchas lighting defaults, zoom defaults, aperture defaults, resolutiondefaults, and/or any other suitable settings. In some embodiments, theimage capture settings are stored in a memory of the IMC MCU 215 (notexplicitly shown).

FIGS. 3A and 3B show front and side views, respectively, of anillustrative portable personal electronic device (PPED) 300, accordingto various embodiments. The PPED 300 can be implemented as a smartphone(e.g., or a laptop computer, tablet computer, wearable device, etc.)with one or more integrated digital cameras 310. For example, asillustrated, the PPED 300 can include a front-facing (e.g., selfie)camera 310 a, a rear-facing camera 310 b (shown in FIG. 3B), a pop-outcamera 310 c, and/or any other suitable integrated cameras 310. At leastone of the integrated cameras 310 is implemented as an immediate-modecamera, such as the IMC system 210 described with reference to FIG. 2A.

Embodiments of the PPED 300 can also include user interface components(e.g., the UI components 105 of FIG. 2A). Some embodiments include oneor more displays 320. Though not explicitly shown, some embodiments ofthe display 320 can have, integrated therewith, capacitive touchscreenelements, another digital camera 310, a fingerprint sensor, and/or othercomponents. The user interface components can also include one or morephysical buttons 330. For example, the physical buttons 330 can includea power button, volume buttons, etc. In some implementations, one ormore of the buttons is dedicated to a particular function, and one ormore of the buttons is dynamically assignable (e.g., by the applicationprocessor 190 and/or other components) to various functions. Forexample, in context of operating a conventional (non-IMC) digitalcamera, a particular button can trigger image acquisition while thecamera application is running in the foreground, while the same buttoncan have other functions while other applications are running in theforeground. Though not shown, the PPED 300 can include additional userinterface components, such as optical sensors, force sensors, biometricsensors, accelerometers, etc.

As described above, certain of the user interface components can beconfigured as always-on components, such that they can respond to userinteractions even while the PPED 300 is locked. Some or all suchalways-on user interface components can be configured as the IMC triggercomponents 205 of FIG. 2A, such that interaction with those componentscan directly trigger image capture by an IMC camera integrated into thePPED 300. In some embodiments, one or more of the physical buttons 330is configured as a IMC trigger component 205. In some such embodiments,the physical button(s) 330 can be coupled (e.g., directly via a hardwareconnection) with the IMC MCU 215 of the IMC system 210. In oneimplementation, a single physical button (e.g., button 330 d, button 330e, etc.) automatically triggers capture of a photo using the IMC system210 (e.g., using camera 310 b). In another implementation, one physicalbutton (e.g., button 330 d) automatically triggers capture of a photousing the IMC system 210, and another physical button (e.g., button 330e) automatically triggers capture of a video using the IMC system 210.In another implementation, concurrently depressing multiple physicalbuttons (e.g., pressing button 330 d and button 330 e together)automatically triggers capture of a photo using the IMC system 210. Inanother implementation, depressing one or more physical buttons for atleast a predetermined amount of time (e.g., holding down button 330 dfor a half-second) automatically triggers capture of a photo using theIMC system 210.

FIGS. 4A-4C show illustrative implementations of PPEDs 300 with varioustypes of IMC trigger components, according to various embodiments.Turning first to FIG. 4A, a PPED 300 c is shown with one or moreintegrated force sensors 410, configured as IMC trigger components. Insome such embodiments, the force sensors 410 can be coupled (e.g.,directly via a hardware connection) with the IMC MCU 215 of the IMCsystem 210. For example, each of one or more force sensors 410 candetect when a user squeezes the PPED 300 c, or otherwise applies atleast a certain amount of force on the PPED 300 c in the location of theforce sensors 410, and can trigger capture of a photo or video using theIMC system 210 in response thereto. In various implementations, the IMCsystem 210 can be triggered by applying force to a particular one of theforce sensors 410, concurrently to multiple force sensors 410, to one ormore force sensors 410 for at least a predetermine amount of time, toone or more force sensors 410 in accordance with a predefined pattern,etc.

Turning to FIG. 4B, a PPED 300 d is shown with touch sensors integratedinto the display (e.g., as part of a capacitive touchscreen), and thetouch sensors are usable as IMC trigger components. In some suchembodiments, some or all of the force sensors can be coupled (e.g.,directly via a hardware connection) with the IMC MCU 215 of the IMCsystem 210. For example, user interaction with one or more touchlocations or patterns 420 can trigger capture of a photo or video usingthe IMC system 210 in response thereto. In various implementations, theIMC system 210 can be triggered by touching one or more particularlocations concurrently, touching one or more particular locations in aparticular order or pattern, tracing out a particular pattern across thetouch sensors, etc.

Turning to FIG. 4C, a PPED 300 e is shown with secondarily activatedtouch sensor locations implemented via the display (e.g., as part of acapacitive touchscreen) and usable as IMC trigger components. Inresponse to the user interacting with one or more IMC trigger components(e.g., buttons, force sensors, touch sensors, etc.), a set of touchregions 430 becomes active. For example, a first touch region 430 abecomes actively associated with video capture, such that subsequentuser interaction with the first touch region 430 a can trigger captureof a video using the IMC system 210; and a second touch region 430 bbecomes actively associated with photo capture, such that subsequentuser interaction with the second touch region 430 b can trigger captureof a photo using the IMC system 210. In some implementations, activationof the touch regions 430 can involve outputting a predefined image tothe display showing indications of the touch regions 430. In otherimplementations, no visual or other indication is provided. For example,the regions are separated by enough distance (e.g., the first touchregion 430 a being the entire top-third of the display, and the secondtouch region 430 b being the entire bottom third of the display), sothat a user can reliably interact with the desired touch region 430without visual cues.

Additionally or alternatively, embodiments of PPEDs 300 can includeother types of IMC trigger components 205. In one such embodiment, photoand/or video capture by the IMC system 210 is triggered by the IMCsystem 210 first capturing a predetermined image or video. For example,as the IMC system 210 is an always-on component, it can be configuredautomatically to respond to detection of a predetermined gesture, facialexpression, etc. In another such embodiment, photo and/or video captureby the IMC system 210 is triggered by an audio input received by amicrophone. For example, a predetermined audio command, recognition of aparticular voice signature, or other audio data can trigger image orvideo capture. In another such embodiment, photo and/or video capture bythe IMC system 210 is triggered by moving the PPED 300 in apredetermined manner (e.g., detected by a gyroscopic sensor, anaccelerometer, etc.). For example, shaking, rotating, waving, orotherwise moving the PPED 300 can trigger image or video capture.

In some embodiments, the PPED 300 can indicate one or more aspects ofIMC system 210 operation. In some such embodiments, capture of image orvideo using the IMC system 210 can trigger an audio response, such as asound of a camera shutter, a ding, or any other suitable audio feedback.In other such embodiments, capture of image or video using the IMCsystem 210 can trigger a visual response, such as output of apredetermined image to the display. For example, during photo or videocapture, or in response to successful photo or video capture, thedisplay can show a graphic (e.g., an image of a camera), text (e.g.,“IMC photo!”), and/or any other visual indication. In someimplementations, certain display components 125 can be implemented foralways-on operation, such as to permit display of a pre-generated image(e.g., a lock screen, an IMC-related graphic, etc.). In otherimplementations, the display components 125 can be triggered to wake upin parallel with image or video capture by the IMC system 210, such thatthe display is ready to output content by the time the image or video iscaptured (or shortly thereafter).

FIG. 5 shows a flow diagram of an illustrative method 500 forimmediate-mode digital image capture, according to various embodiments.Embodiments of the method 500 can operate in context of animmediate-mode camera (IMC) system of a portable personal electronicdevice (PPED), such as those described above with reference to FIGS.2A-4C. Embodiments of the method 500 can begin at stage 504 bydetecting, by an IMC controller of an IMC system integrated into a PPED,an interaction with an IMC trigger component. The IMC trigger componentis coupled with the IMC controller and is one of a set of always-on userinterface components integrated into the PPED. For example, the IMCtrigger component is a physical button, a force sensor, a touch sensor,etc. In some embodiments, the detecting at stage 504 includes detectingconcurrent interaction with a plurality of IMC trigger components (e.g.,pressing two physical buttons at substantially the same time). In otherembodiments, the detecting at stage 504 includes detecting apredetermined pattern of interaction with one or more IMC triggercomponents (e.g., tapping a certain number of times and/or in a certainrhythm, swiping along a certain path, etc.).

At stage 508, embodiments can trigger, by the IMC controller, inresponse to the interaction at stage 504, capture of image data by adigital camera of the IMC system. In some embodiments, the detecting atstage 504 is performed while the application processor is in a sleepmode (e.g., the PPED is locked). In some such embodiments, the detectingat stage 504 and the triggering at stage 508 can be performed withoutwaking the application processor out of the sleep mode (e.g., withoutunlocking the PPED). In other such embodiments, the method 500 canfurther trigger, by the IMC controller, in response to the interaction,in parallel with triggering the capture of the image data (e.g., at thesame time, in close temporal proximity to, etc.), wakeup of theapplication processor from the sleep mode. For example, even though thecapture of the image data and the waking of the application occur inparallel, the capture of the image data can occur prior to completingwakeup of the application processor.

In some embodiments, the IMC controller is in communication with amemory having, stored thereon prior to the detecting, image capturesettings. In such embodiments, the triggering at stage 508 can be suchthat the capture of the image data is in accordance with the imagecapture settings (e.g., the image data is captured automatically inaccordance with preset defaults). In other such embodiments, prior tothe detecting, the method 500 can include receiving user interactionsindicating a user configuration of one or more of the image capturesettings; and updating the one or more of the image capture settings inthe memory, by the IMC controller, responsive to and in accordance withthe user interactions.

At stage 512, embodiments can pretreat the image data by the IMCcontroller to generate pretreated image signals. At stage 516,embodiments can communicate the pretreated image signals via aninterface bus to an always-on region of an application processorintegrated into the PPED. For example, the image data can be pretreatedfor communication via a specific implementation of interface bus, suchas a serial peripheral interface. In some embodiments, the applicationprocessor has, implemented thereon, a trusted execution environment, andthe trusted execution environment has the always-on region implementedtherein.

It will be understood that, when an element or component is referred toherein as “connected to” or “coupled to” another element or component,it can be connected or coupled to the other element or component, orintervening elements or components may also be present. In contrast,when an element or component is referred to as being “directly connectedto,” or “directly coupled to” another element or component, there are nointervening elements or components present between them. It will beunderstood that, although the terms “first,” “second,” “third,” etc. maybe used herein to describe various elements, components, these elements,components, regions, should not be limited by these terms. These termsare only used to distinguish one element, component, from anotherelement, component. Thus, a first element, component, discussed belowcould be termed a second element, component, without departing from theteachings of the present invention. As used herein, the terms “logiclow,” “low state,” “low level,” “logic low level,” “low,” or “0” areused interchangeably. The terms “logic high,” “high state,” “highlevel,” “logic high level,” “high,” or “1” are used interchangeably.

As used herein, the terms “a”, “an” and “the” may include singular andplural references. It will be further understood that the terms“comprising”, “including”, having” and variants thereof, when used inthis specification, specify the presence of stated features, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, steps, operations,elements, components, and/or groups thereof. In contrast, the term“consisting of” when used in this specification, specifies the statedfeatures, steps, operations, elements, and/or components, and precludesadditional features, steps, operations, elements and/or components.Furthermore, as used herein, the words “and/or” may refer to andencompass any possible combinations of one or more of the associatedlisted items.

While the present invention is described herein with reference toillustrative embodiments, this description is not intended to beconstrued in a limiting sense. Rather, the purpose of the illustrativeembodiments is to make the spirit of the present invention be betterunderstood by those skilled in the art. In order not to obscure thescope of the invention, many details of well-known processes andmanufacturing techniques are omitted. Various modifications of theillustrative embodiments, as well as other embodiments, will be apparentto those of skill in the art upon reference to the description. It istherefore intended that the appended claims encompass any suchmodifications.

Furthermore, some of the features of the preferred embodiments of thepresent invention could be used to advantage without the correspondinguse of other features. As such, the foregoing description should beconsidered as merely illustrative of the principles of the invention,and not in limitation thereof. Those of skill in the art will appreciatevariations of the above-described embodiments that fall within the scopeof the invention. As a result, the invention is not limited to thespecific embodiments and illustrations discussed above, but by thefollowing claims and their equivalents.

What is claimed is:
 1. A portable personal electronic device (PPED) forimmediate-mode digital image capture, the PPED comprising: anapplication processor configured selectively to operate in a sleep modeor an active mode, the application processor having an always-on regionthat remains active when the application processor is in the sleep mode;a plurality of user interface components having one or more always-onuser interface components, at least one of the one or more always-onuser interface components being an IMC trigger component; and animmediate-mode camera (IMC) system having a digital camera assemblycontrolled by an IMC controller, the IMC controller coupled with andresponsive to the IMC trigger component, the IMC controller configuredto: detect an interaction with the IMC trigger component; trigger, inresponse to detecting the interaction, capture of image data by thedigital camera assembly; pretreat the image data to generate pretreatedimage signals; and communicate the pretreated image signals via aninterface bus to the always-on region of the application processor. 2.The PPED of claim 1, wherein: the IMC controller is to detect theinteraction while the application processor is in the sleep mode; andthe IMC controller is to perform the detecting and the triggeringwithout waking the application processor out of the sleep mode.
 3. ThePPED of claim 1, wherein the detecting is while the applicationprocessor is in a sleep mode, and further comprising: triggering, by theIMC controller, in response to the interaction, in parallel withtriggering the capture of the image data, wakeup of the applicationprocessor from the sleep mode, wherein the capture of the image dataoccurs prior to completing wakeup of the application processor.
 4. ThePPED of claim 1, wherein: the IMC system further has a memory having,stored thereon, image capture settings; and the IMC controller is toperform the triggering such that the capture of the image data is inaccordance with the image capture settings.
 5. The PPED of claim 4,wherein the IMC controller is further to: update one or more of theimage capture settings in the memory responsive to and in accordancewith user interactions with one or more of the user interface componentsindicating user configuration of the one or more of the image capturesettings.
 6. The PPED of claim 1, wherein the IMC trigger component isdirectly coupled with the IMC controller via a hardware coupling.
 7. ThePPED of claim 1, wherein the interface bus comprises a serial peripheralinterface.
 8. The PPED of claim 1, wherein the application processorhas, implemented thereon, a trusted execution environment, the trustedexecution environment comprising the always-on region.
 9. The PPED ofclaim 1, wherein the IMC trigger component is at least one of: aphysical button, a force sensor, or a touch sensor.
 10. The PPED ofclaim 1, wherein the PPED is a smartphone.
 11. A method forimmediate-mode digital image capture, the method comprising: detecting,by an immediate-mode camera (IMC) controller of an IMC system integratedinto a portable personal electronic device (PPED), an interaction withan IMC trigger component coupled with the IMC controller, the IMCtrigger component being one of a set of always-on user interfacecomponents integrated into the PPED; triggering, by the IMC controller,in response to the interaction, capture of image data by a digitalcamera of the IMC system; pretreating the image data by the IMCcontroller to generate pretreated image signals; and communicating thepretreated image signals via an interface bus to an always-on region ofan application processor integrated into the PPED.
 12. The method ofclaim 11, wherein: the detecting is while the application processor isin a sleep mode; and the detecting and the triggering are performedwithout waking the application processor out of the sleep mode.
 13. Themethod of claim 11, wherein the detecting is while the applicationprocessor is in a sleep mode, and further comprising: triggering, by theIMC controller, in response to the interaction, in parallel withtriggering the capture of the image data, wakeup of the applicationprocessor from the sleep mode, wherein the capture of the image dataoccurs prior to completing wakeup of the application processor.
 14. Themethod of claim 11, wherein: the IMC controller is in communication witha memory having, stored thereon prior to the detecting, image capturesettings; and the triggering is such that the capture of the image datais in accordance with the image capture settings.
 15. The method ofclaim 14, further comprising, prior to the detecting: receiving userinteractions indicating a user configuration of one or more of the imagecapture settings; and updating the one or more of the image capturesettings in the memory, by the IMC controller, responsive to and inaccordance with the user interactions.
 16. The method of claim 11,wherein: the detecting the interaction with the IMC trigger componentcomprises detecting concurrent interaction with a plurality of IMCtrigger components.
 17. The method of claim 11, wherein: the detectingthe interaction with the IMC trigger component comprises detecting apredetermined pattern of interaction with one or more IMC triggercomponents.
 18. The method of claim 11, wherein the IMC triggercomponent is directly coupled with the IMC controller via a hardwarecoupling.
 19. The method of claim 11, wherein the interface buscomprises a serial peripheral interface.
 20. The method of claim 11,wherein the application processor has, implemented thereon, a trustedexecution environment, the trusted execution environment comprising thealways-on region.